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ABSTRACT 


A  considerable  amount  of  money  and  resources  has  been  spent  on  the 
development  of  the  new  programming  language  Ada.  The  University  of 
Maryland  and  General  Electric  have  studied  the  development  of  a  software 
project  written  in  Ada.  This  paper  presents  the  analysis  of  the  effort, 
change,  and  error  data.  The  total  effort  spent  on  training  and  metho¬ 
dology  was  20%  of  the  total  effort  spent  on  the  project;  this  was  more 
than  the  effort  spent  on  any  other  phase.  The  greatest  error  rates 
appeared  to  be  associated  with  the  most  Ada-specific  features:  tasking, 
generics  and  compilation  units.  Experience  with  high-level  languages 
seemed  to  be  associated  with  a  better  ability  to  grasp  Ada  concepts. 
Finally,  the  results  strongly  indicate  the  need  for  support  tools  for  an 
Ada  programming  environment. 
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1.  INTRODUCTION 


\ 

The  Department  of  Defense  has  spent  a  considerable  amount  of  money 
and  resources  on  the  development  of  the  new  programming  language  Ada. 

To  develop  a  better  understanding  of  the  nature  of  this  language,  it  is 
necessary  to  pull  it  out  of  the  research  arena  and  use  it  in  an  indus¬ 
trial  environment  where  one  must  deal  with  Issues  such  as  training, 
budgets  and  support  facilities. 


To  gain  insight  into  the  use  of  this  new  language,  the  University 
of  Maryland  and  General  Electric  have  jointly  undertaken  a  study  of  the 
development  of  a  software  project  written  in  Ada.v  A  subset  of  a  previ¬ 
ously  written  satellite  ground  control  system  wa^to  be  reimplemented  by 
four  GE  programmers  over  a  period  of  approxim^efy  one  year  using  Ada. 
The  process  involved  two  separate  groupsjuHT  a  team  consisting  of  four 
programmers  whose  task  wasjLO-build'  the" software  and  to  view  this  as  a 
"normal"  lndustrj^il'-t:asir'to  as  great  an  extent  as  possible,  and  (2)  a 
team  consi^tirtgof  individuals  from  GE  and  the  University  of  Maryland 
who^>(otild  observe  the  programmers  with  a  minimal  amount  of  interference. 

—  A  set  of  goals  and  questions  was  established  at  the  beginning  of 
the  project.  These  included  generic  goals  for  any  software  development 
project,  goals  relating  to  Ada  as  a  design  and  Implementation  language, 
and  goals  relating  to  metrics  for  the  Ada  Programming  Support  Environ¬ 
ment  (APSE).  Data  collected  from  the  project  were  analyzed.  This  paper 
describes  the  observations  which  provide  answers  to  a  subset  of  the 
goals  and  questions.  While  some  of  the  aunswers  are  relevant  only  to  our 
particular  environment,  others  apply  to  any  group  wishing  to  use  Ada  for 
the  first  time,  and  still  others  apply  to  auiy  Ada  environment.  More 
specifically,  this  paper  addresses  those  goals  that  are  related  to 
effort,  changes,  errors  and  programmer  characteristics.  ^ 


2.  BACKGROUND 


The  project  under  study  Involved  the  redesign  and  implementation  in 
Ada  of  a  portion  of  a  satellite  ground  control  system  originally  written 
in  FORTRAN.  This  subset  included  an  interactive  operator  interface, 
graphic  output  routines,  and  concurrent  telemetry  monitoring. 

Four  programmers  with  diverse  backgrounds  were  selected  for  this 
project  in  order  to  determine  whether  a  programmer's  experience  and  edu¬ 
cation  will  influence  his  understanding  and  use  of  Ada.  Table  2.1  shows 
the  education  and  experience  of  each  programmer  and  the  programming 
languages  each  is  familiar  with.  The  lead  programmer  was  fluent  in  FOR¬ 
TRAN  and  assembler  languages.  He  had  many  years  of  industrial  experi¬ 
ence  working  in  the  application  area.  The  senior  programmer  had  a 
master's  degree  in  computer  science  and  had  worked  with  many  languages, 
such  as  COBOL,  PL/1,  Lisp,  ALGOL,  and  SNOBOL  in  addition  to  FORTRAN  and 


assembler.  He  had  a  reasonable  amount  of  experience  with  the  applica¬ 
tion  area,  though  not  nearly  as  extensive  as  the  lead  programmer.  The 
Junior  prograunmer  had  just  obtained  a  bachelor's  degree  in  computer  sci¬ 
ence  and  had  programmed  in  Pascal  as  well  as  all  the  languages  that  the 
senior  programmer  knew.  He  had  no  Industrial  experience  whatsoever  and 
was  not  at  all  familiar  with  the  application  area.  The  librarian's  com¬ 
puter  science  background  consisted  of  a  single  course  in  FORTRAN  pro¬ 
gramming. 


None  of  the  programmers  knew  Ada  before  the  project  began.  They 
received  one  month  of  intensive  training  in  Ada.  They  viewed  fifteen 
hours  of  videotaped  lectures  given  by  Ichbiah,  Firth,  and  Barnes  (the 
major  developers  of  the  language)  over  a  period  of  four  days.  This  was 
followed  by  six  days  of  further  training  given  by  George  W.  Cherry  of 
Language  Automation  Associates  which  was  spread  over  a  period  of  four 
weeks.  During  this  time,  the  programmers  also  practiced  writing  Ada 
programs,  read  the  Ada  reference  manual  and  reviewed  their  class  notes. 
The  NYU  Ada/Ed  interpreter  was  used  for  programming  assignments  which 
Included  a  500-line  team  project.  Finally,  the  programmers  were  given  a 
half-day  class  on  software  engineering  techniques  by  Victor  R.  Basil!  of 
the  University  of  Maryland.  Among  the  topics  discussed  were  chief- 
programmer  teams,  design  and  code  walkthroughs,  and  structured  program¬ 
ming. 

The  programmers  never  saw  the  comparable  Fortran  source  programs. 

It  was  estimated  that  the  equivalent  Fortran  subset  was  about  10,000 
lines  of  Fortran  code  -  a  size  amenable  for  the  programmers  to  build  in 
a  year. 

Table  2.1  -  Backgrounds  of  Programmers 


Programmer 

Years  of 
Professional 
Experience 

Education 

Languages 

Lead 

9 

B.S. 

(Comp.  Sci.) 

FORTRAN,  Assembler 

Senior 

7 

M.S. 

(Comp.  Sci.) 

FORTRAN,  Assembler, 
COBOL,  PL/1,  Lisp, 
ALGOL,  SNOBOL 

Junior 

0 

B.S. 

(Comp.  Sci.) 

FORTRAN,  Assembler, 
COBOL,  PL/1,  Lisp, 
ALGOL,  SNOBOL,  Pascal 

Librarian 

0 

High  School 
Degree 

FORTRAN 

The  project  began  in  February  1982  and  ended  in  July  1983. 
Requirements  amalysis  of  this  project  was  done  concurrently  with  train¬ 
ing  during  the  first  month.  Following  this,  system  design  using  an 
Ada-like  Programming  Design  Language  (PDL).  coding,  and  testing  were 
done. 


3.  GOALS 


This  study  attempts  to  answer  goals  and  questions  from  four  dif¬ 
ferent  areas  of  the  Ada  project.  We  characterized  the  effort,  the 
changes,  and  the  errors.  We  also  tried  to  associate  each  programmer's 
background  with  his  use  of  Ada  and  his  performance  on  the  project. 

In  order  to  collect  the  data  necessary  for  the  study,  the  program¬ 
mers  were  asked  to  complete  various  forms.  The  forms  relevant  to  this 
study  are: 

a)  component  status  report  forms 

b)  change  request  forms 

c)  error  description  forms 

d)  individual  document  change  report  forms 

A  component  status  report  form  contains  Information  on  the  weekly  break¬ 
down  of  effort  spent  by  a  programmer  on  each  phase  of  the  project.  Bach 
time  a  need  for  change  was  determined,  a  change  request  form  was  filled 
out.  If  the  change  was  an  error  correction,  an  error  description  form 
was  also  completed.  An  Individual  document  change  report  form  was 
filled  out  for  every  component  Involved  in  a  change.  Samples  of  these 
forms  may  be  found  in  Appendix  7. 

In  addition  to  the  data  collected  on  the  forms,  a  copy  of  the  last 
design  and  code  versions  for  each  module  was  kept.  The  source  code 
measures  used  in  this  paper  were  taken  from  the  latest  version  for  all 
modules . 


3.1  Effort 


In  order  to  analyze  the  effort  in  the  project,  we  determined  how 
the  effort  was  distributed  over  the  phases  (i.e.  requirements,  design, 
coding,  testing,  training,  etc.)  of  the  project.  In  addition,  we  deter¬ 
mined  how  the  effort  for  the  project  was  distributed  over  time. 


3.2  Changes 


Our  second  goal  was  to  analyze  the  changes  In  the  project.  We  cal¬ 
culated  the  number  of  each  type  of  change,  where  the  types  of  changes 
are: 

a)  error  corrections 

b)  changes  In  problem  domain 

c)  planned  enhancements 

d)  avoidances  of  apparent  problems  with  the  compiler 

e)  avoidances  of  other  problems  in  the  developing  environment 

f)  adaptations  to  a  change  in  the  developing  environment 

g)  improvements  of  clarity,  maintainability  or  documentation 

h)  optimization  of  time,  space  or  accuracy 

1)  insertion  or  deletion  of  debug  code 

j)  other  than  above 

Furthermore,  the  number  of  changes  according  to  document  type  was  tabu¬ 
lated  based  on  the  highest  level  document  modified  in  any  component  for 
each  change.  The  document  types  are  requirements,  PDL,  and  code 
modules . 

Another  aspect  of  the  analysis  involved  determining  how  the  changes 
were  distributed  over  the  software  development  cycle.  We  calculated  the 
number  of  components  Involved  in  each  change  and  the  number  of  Interface 
changes  made.  Finally,  we  determined  how  long  it  took  to  establish  the 
need  for  change  and  how  long  it  took  to  design  and  implement  the  change. 
From  these  results,  we  hoped  to  answer  other  questions  such  as  how 
effective  Ada  is  in  producing  software  that  is  easily  changed  and 
whether  the  need  for  change  was  easily  determined. 


3.3  Errors 


Our  third  major  goal  was  to  characterize  the  errors  that  resulted. 
We  wanted  to  determine  which  of  the  following  types  of  errors  occurred: 

a)  requirements  Incorrect 

b)  requirements  misinterpreted 

c)  design  Incorrect 

d)  design  misinterpreted 

e)  code  Incorrect 

f)  external  environment  misunderstood  (not  language  or  compiler) 

g)  clerical  error 

There  were  specific  questions  that  we  tried  to  answer  regarding  the 
errors.  What  activities  were  used  to  detect  and  isolate  the  errors? 
Were  they  easy  to  find  and  to  correct?  How  much  effort  was  required  to 
correct  them?  Where  was  the  information  needed  to  correct  errors  found? 
At  which  stage  in  the  project  did  they  enter  the  system?  For  each  type 
of  error,  we  determined  the  highest  level  document  that  needed  to  be 
changed.  We  also  obtained  the  number  of  interface  errors  and  the  dis¬ 
tribution  of  errors  over  time. 


The  errors  were  classified  as  follows: 

a)  Language 

b)  Problem 

c)  Clerical 

Language  errors  were  those  which  involved  the  syntax  or  semantics  of  a 
feature  or  those  which  involved  the  concept  behind  a  feature.  The  prob¬ 
lem  .category  encompassed  logic  errors  and  errors  related  to  the  environ¬ 
ment.  Clerical  errors  Included  those  due  to  carelessness,  e.g.  typo¬ 
graphical  errors . 

We  determined  which  features  of  Ada  are  commonly  involved  In 
errors.  We  also  summarized  the  programmers'  responses  to  questions 
regarding  their  understanding  of  the  features,  such  as:  Does  the  docu¬ 
mentation  explain  the  features  clearly?  Can  the  errors  be  attributed  to 
lack  of  understanding  of  Ada?  lack  of  experience  with  a  feature?  confu¬ 
sion  with  another  language? 

We  tried  to  determine  If  the  use  of  Ada  PDL  causes  a  preoccupation 
with  syntax  during  the  design  stage.  Furthermore,  we  attempted  to  find 
out  if  there  were  errors  uncovered  during  the  design  stage  that  ordi¬ 
narily  would  have  been  found  during  coding  because  of  the  use  of  Ada 
PDL. 


3.4  Programmers 


Our  fourth  major  goal  was  to  determine  whether  there  was  any  rela¬ 
tionship  between  the  background  of  the  programmers  Involved  and  their 
use  of  Ada.  We  obtained  breakdowns  of  effort,  changes,  and  errors  by 
programmer.  Are  certain  types  of  errors  associated  with  particular  pro¬ 
grammers?  Are  some  features  of  the  language  used  incorrectly  or  inap¬ 
propriately  by  the  programmers?  Finally,  we  tried  to  detennine  if  pro¬ 
grammers  with  no  or  little  previous  high-level  lamguage  experience  have 
more  or  fewer  problems  with  Ada  than  prograunmers  with  substantial  high- 
level  language  experience. 


4.  OBSERVATIONS 


4.1  FACTORS 

Several  factors  affected  the  outcome  of  this  study,  and  It  Is 
Important  to  understand  them  so  the  results  of  the  analysis  can  be 
interpreted  properly.  First  of  all,  no  full  production  quality  proces¬ 
sor  was  available  at  the  time  the  experiment  was  being  conducted,  and 
this  had  a  major  Impact  on  the  project.  No  compiler  was  available  ini¬ 
tially,  and  several  rumored  products  were  "promised"  imminently. 
Finally,  it  became  apparent  that  no  such  compiler  would  be  made 


5 


available,  so  the  useful,  but  very  slow,  NYU  Ada/Ed  interpreter  was 
used.  However,  even  that  became  unusable  towards  the  end  of  the  project 
as  the  size  of  the  developing  system  grew.  This  had  a  demoralizing 
effect  on  the  programmers  who  did  not  finish  coding  or  testing  the  pro¬ 
ject.  VUien  the  ROLM  compiler  became  available,  some  further  testing  was 
done.  The  results  in  this  paper  are  based  on  data  collected  through 
coding  and  some  unit  testing.  The  effort  data  used  in  this  study  show 
that  little  time  was  spent  on  testing.  (See  Table  4.1a.)  In  addition, 
the  vast  majority  of  the  Ada-related  errors  were  syntax  errors.  This 
might  not  have  been  the  case  if  the  code  had  actually  been  executed  and 
testing  had  been  completed.  It  is  probable  that  many  more  logic  errors 
would  have  been  uncovered  had  syntax  error-free  compilation  of  all  the 
modules  been  achieved. 

Secondly,  the  lack  of  an  automated  PDL  processor  prevented  the  pro¬ 
grammers  from  investigating  deeper  logic  issues  instead  of  simple  syntax 
errors.  Many  errors  which  first  appeared  in  the  design  stage  would  have 
been  caught  then  had  a  PDL  processor  been  used.  However,  they  were  not 
caught  until  compilation  during  the  coding  stage.  In  several  of  these 
cases,  the  code  was  changed  but  the  programmers  failed  to  correct  the 
design  document. 

Thirdly,  even  though  the  requirements  were  taken  from  a  previous 
FORTRAN  project,  only  a  subset  of  the  project  was  to  be  recoded  in  Ada. 
Thus,  the  requirements  document  had  to  be  cut  down  accordingly.  The 
resulting  set  of  requirements  was  improved  and  made  into  a  consistent 
subsystem.  This  accounts  for  the  effort  spent  on  requirements  and  the 
substantial  number  of  requirements  changes.  (See  Tables  4.1a  and  4.2a.) 

Appendix  1  shows  the  effort  spent  on  coding  throughout  the  project. 
There  was  a  'false  start'  in  the  12th  week  of  the  project.  Since  the 
programmers  had  a  problem  with  methodology,  in  particular,  writing 
abstract  PDL,  they  began  coding  the  project  before  they  were  prepared  to 
do  so.  They  were  told  to  stop,  and  they  did  not  resume  coding  until  the 
25th  week  of  the  project.  This  accounts  for  the  large  gap  of  no  coding 
activity  for  nine  weeks  in  the  middle  of  the  figure. 

The  project  was  temporarily  discontinued  after  the  49th  week,  and 
further  testing  was  resumed  in  the  62nd  week  by  the  junior  programmer. 
This  explains  the  periods  of  no  activity  in  the  graphs  of  effort, 
changes  and  errors  over  time  contained  in  the  appendices. 

A  final  factor  which  influenced  the  results  was  that  Ada  is  a  com¬ 
pletely  new  language  with  features  not  present  in  any  other  programming 
language.  Thus,  20l>  of  the  total  effort  was  spent  on  training  and 
methodology,  which  is  more  than  the  effort  spent  on  any  other  phase  of 
the  project.  (See  Table  4.1a.)  This  is  a  much  higher  percentage  than 
would  typically  be  spent  on  most  projects.  Furthermore,  even  this 
amount  seems  insufficient  because  the  progrsunmers  indicated  that  they 
did  not  feel  comfortable  with  Ada  until  after  they  left  the  project. 


4.2  DISCUSSION 
4.2.1  Effort 
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Table  4.1a  shows  the  amount  of  effort  spent  on  each  phase  of  the 
project.  As  stated  previously,  training  and  methodology  accounted  for 
20%  of  the  effort,  a  very  high  percentage  compared  to  most  other  pro¬ 
jects.  The  distribution  of  effort  over  time  follows  normal  software 
development  patterns  euid  is  shown  in  Appendix  1.  Productivity  was  meas¬ 
ured  and  is  presented  in  Table  4.1b.  It  was  determined  that  18.52  lines 
of  code  (including  comments  but  excluding  blank  lines)  and  9«83  lines  of 
text  (lines  containing  part  of  an  Ada  statement)  were  developed  per  hour 
spent  in  code  development.'  Productivity  for  the  effort  spent  on  the 
entire  project  was  also  calculated,  but  the  values  are  upper  bounds  (and 
may  not  be  meaningful)  since  the  project  was  not  completed. 


Table  4.1a  -  Effort  for  Each  Phase  of  the  Project 


Project  Phase 


Amount  Of  Time 
(in  hours) 


Percentage 


Requirements  Analysis 
Requirements  Writing 
Design  Creation 
Design  Reading 
Formal  Design  Review 
Coding 
Code  Reading 
Formal  Coding  Review 
Unit  Testing 
Integration  Testing 
Review  Testing 
Training  and  Methodology 

Other  Activity 

Total  Requirements 
Total  Design 
Total  Code  Development 
Total  Testing 

Total  Training  and  Methodology 
Total  Other  Activity 

Entire  Project 


530.5 

113.6 

514.4 
37.7 

162.4 

305.6 

13.3 

62.3 

332.7 
0 

0 

849.1 
1245.7 

644.1 

714.5 

381.2 
332.7 
849.1 

1245.7 

4167.3 


73 
73 
34 
91 
3.89% 
7.33% 
0.32% 
1.50% 
7.98% 
0.00% 
0.00% 
20.38% 

29.89% 

15.46% 

17.14% 

9.15% 

7.98% 

20.38% 

29.89% 

100.00% 


Table  4.1b  -  Productivity 


Code  Developed 
Text 

All  Non-blank  Lines 


Productivity 
(Lines  of  Code  per  Hour) 
Total  Code  Entire  Project 
Development 

9T^3  1.09 

18.52  2.04 


Note  -  Text  refers  to  any  line  containing  part  of  an  Ada  statement 


4.2.2  Changes 

The  analysis  of  the  337  change  request  forms  and  the  439  individual 
document  change  forms  resulted  in  the  following  observations.  The 
number  of  changes  according  to  document  type  (the  highest  level  of  docu¬ 
ment  modified  in  any  component  for  this  change)  is  displayed  in  Table 
4.2a.  Code  changes  accounted  for  6l%  of  the  changes.  As  stated  previ¬ 
ously  though,  many  of  these  changes  were  errors  which  should  have  been 
caught  at  the  design  stage.  Thirty- two  percent  of  the  changes  were  in 
design  documents,  and  only  seven  percent  were  in  requirements  documents. 
The  number  of  overall  changes  and  the  number  of  non-error  changes  per 
thousand  lines  of  code  are  presented  in  Tables  4.2b  and  4.2c  respec¬ 
tively.  These  numbers  are  lower  bounds  and  may  not  be  meaningful 
because  the  project  was  not  completed. 

The  breakdown  by  type  of  change  is  shown  in  Table  4.3.  The  major¬ 
ity  (57$)  of  the  changes  were  error  corrections  which  will  be  described 
in  detail  later.  Of  the  non-error  changes,  52$  were  improvements  of 
clarity,  maintainability  and  documentation.  Most  of  these  were  PDL  and 
requirements  changes  as  shown  in  Appendix  2j  this  indicates  that  there 
was  concern  for  good  design. 

The  time  to  determine  the  need  for  change  was  one  hour  or  less  in 
almost  all  cases  as  shown  in  Table  4.4.  In  addition,  46$  required  only 
one  tenth  of  an  hour.  This  indicates  that  the  need  for  these  changes 
was  easily  determined.  There  were  a  few  changes  which  took  much  longer 
than  the  average  of  one  half  hour  per  change.  It  took  one  or  more  days 
to  determine  the  need  for  each  of  two  planned  enhancements.  This  was 
reasonable  for  the  type  of  change.  A  different  code  change  which  took 

Table  4.2a  -  Changes  by  Document  Type 


Document  Type 

Number  of  Changes 

Percentage 

Requirements 

24 

tTTB 

PDL 

107 

31.75$ 

Code  Module 

206 

61.13$ 

Q 

tfEII 


Table  4.2b  -  Number  of  Changes  per  Thousand  Lines  of  Code 

Changes  in  Code  All  Changes 
_  Modules  Only 

Text  44  71 

Non-blank  lines  23  38 


Table  4.2c  -  Number  of  Non-Error  Changes  per  Thousand  Lines  of  Code 

Changes  in  Code  All  Changes 
_  Modules  Only 

Text  12  31 

Non-blank  lines  7  16 


31^ 


Table  4.3  -  Breakdown  of  Changes  by  Type 


Type  of  Change 

Number  of  Changes 

Percentage 

Error  Corrections 

192 

56.96% 

Ch8mges  in  Problem  Domain 

1 

0.29% 

Planned  Enhancements 

9 

2.67% 

Avoidances  of  Apparent  Problems 

•  18 

5.37% 

with  the  Compiler 

Avoidances  of  Other  Problems  in 

2 

0.59% 

the  Developing  Environment 

Adaptations  to  a  Change  in  the 

7 

2.08% 

Developing  Environment 

Improvements  of  Clarity,  Maintain- 

76 

22.55% 

ability  or  Documentation 

Optimization  of  Time,  Space  or 

2 

0.59% 

Accuracy 

Insertion  or  Deletion  of  Debug  Code 

9 

2.67% 

Other  Than  Above 

21 

6.23% 

two  days  involved  avoiding  a  problem  with  the  compiler.  Another  change 
which  involved  a  global  definitions  package  took  two  days.  It  entailed 
the  detailed  modification  and  interfacing  of  several  components. 

Surprisingly,  the  amount  of  time  needed  to  design  and  implement 
changes  was  very  small.  (See  Table  4.5.)  Again,  the  vast  majority  of 


Table  4.4  -  Time  to  Determine  Need  for  Change 


Effort 

Number  of  changes 

Percentage 

0.1  hour 

155 

46.00% 

0.2  hour 

81 

24.04% 

0.3  hour 

23 

6.82% 

0.4  hour 

2 

0.59% 

0.5  hour 

36 

10.68% 

0.6  hour 

5 

1.48% 

0.8  hour 

9 

2.67% 

0.9  hour 

3 

0.89% 

1  hour 

11 

3.26% 

2  hours 

2 

0.59% 

3  hours 

1 

0.30% 

4  hours 

4 

1.19% 

6  hours 

1 

0.30% 

1  day 

1 

0.30% 

2  days 

3 

0.89% 

Total  Effort  =  166.3  hours 
Mean  =  0.49  hours  per  change 
Standard  Deviation  =  1.64  hours  per  change 
Median  =  0.20  hours  per  change 


changes  took  one  hour  or  less  to  handle.  Of  the  code  changes,  all 
except  five  took  two  hours  or  less.  Two  changes,  which  took  three  hours 
and  one  day  respectively,  involved  avoiding  problems  with  the  compiler. 
One  change  which  took  one  and  a  half  days  resulted  from  an  adaptation  to 
a  change  in  the  development  environment.  One  code  change  which  took 
four  hours  was  an  error  correction  and  will  be  discussed  in  the  errors 
section.  One  code  change  took  four  days;  this  Involved  the  same  global 
definitions  package  as  described  above.  The  few  other  changes  which 
took  much  longer  than  usual  were  mostly  planned  enhancements  and 
improvements  of  clarity,  maintainability  and  documentation  of  require¬ 
ments  documents.  The  change  which  took  one  week  was  a  planned  enhance¬ 
ment  in  a  requirements  section. 

Table  4.5  -  Time  to  Design  and  Implement  Changes 


Effort 

Number  of  Changes 

Percentage 

0.1  hour 

164 

48.66) 

0.2  hour 

78 

23.14) 

0.3  hour 

19 

5.63) 

0.4  hour 

14 

4.14) 

0.5  hour 

15 

4.44) 

0.6  hour 

7 

2.08) 

0.7  hour 

2 

0.59) 

0.8  hour 

0 

0.00) 

0.9  hour 

2 

0.59) 

1  hour 

10 

2.97) 

1 . 1  hours 

2 

0.59) 

1.2  hours 

1 

0.30) 

1.3  hours 

0 

0.00) 

1 . 4  hours 

1 

0.30) 

1.5  hours 

4 

1.19) 

2  hours 

4 

1.19) 

3  hours 

1 

0.30) 

3.5  hours 

1 

0.30) 

4  hours 

1 

0.30) 

5  hours 

1 

0.30) 

5.2  hours 

1 

0.30) 

6  hours 

1 

0.30) 

0.8  day 

1 

0.30) 

1  day 

2 

0.59) 

1 . 5  days 

1 

0.30) 

2  days 

1 

0.30) 

3  days 

1 

0.30) 

4  days 

1 

0.30) 

1  week 

1 

0.30) 

Total  Effort  =  260.1  hours 
Mean  =  0.77  hours  per  change 
Standard  Deviation  =  3 *34  hours  per  change 
Median  =  0.20  hours  per  change 


The  total  effort  spent  on  determining  the  need  for  and  implementing 
changes  was  426.4  hours,  which  is  10jl  of  the  total  effort  for  the  entire 
project.  The  average  coat  was  1.27  hours  per  change.  However,  it 
should  be  noted  that  most  of  the  changes  took  much  less  time  than  this. 
The  changes  began  in  the  nineteenth  week  of  the  project  and  the  distri¬ 
bution  of  the  changes  throughout  the  software  development  cycle  is  shown 
in  Appendix  3> 

The  number  of  components  involved  in  each  change  is  shown  in  Table 
4.6.  Only  one  component  was  modified  in  77%  of  the  changes,  but  up  to 
five  components  were  involved  in  some  changes.  We  also  determined  the 
number  of  interface  changes.  (See  Tables  4.7a  and  4.7b.)  In  this  paper, 
an  Interface  change  is  defined  as  one  which  entails  a  change  in  more 
than  one  component  at  the  same  level  of  document.  There  was  a  total  of 
70  interface  changes  (21$  of  all  changes).  Only  2.9$  of  these  were  in 
the  requirements,  and  the  rest  were  equally  divided  between  design  and 
code.  As  many  as  five  components  were  involved  in  these  interface 
changes . 


Table  4.6  -  Number  of  Components  Involved  in  Each  Change  by  Type  of  Do¬ 
cument 

Number  of  Changes  by 
Type  of  Document 

Number  of  Req.  PDL  Code  All  Levels 

Components  Involved  of  Document  » 

1  23  t~~S2  i77~f  2S0 

2  2  19  24  47 

3  0  7  7  17 

4  0  6  3  10 

_ 5 _ 0  2  0 _ 3 

*  The  first  three  columns  do  not  add  up  to  the  last  column.  This  is  be¬ 
cause  a  change  may  Involve  several  components  at  different  levels  of  do¬ 
cument.  (e.g.  one  change  may  Involve  three  components  -  two  PDL  com¬ 
ponents  and  one  code  component . ) 

Table  4.7a  -  Interface  Changes 


Total 


2 

34 

34 


Number 

2 

Req. 

2 

PDL 

19 

Code 

24 

Total 

45 

$ 

64.28$ 

2 

2.86$ 


48.57$ 

100.00$ 


100.00$ 


Table  4. 7b  -  Non-Error  Interface  Changes 


Number 

of  Components  Involved 

2 

3  4  5 

Total 

* 

Req. 

2 

0 

PDL 

14 

3 

Code 

14 

7 

Total 

30 

10 

* 

58.82* 

19.61* 

2 

3.92* 


4.2.3  Errors 


A  total  of  192  error  description  forms  were  examined.  Each  of 
these  forms  corresponds  to  a  change  request  which  falls  under  the  error 
correction  category.  Table  4.8  shows  a  breakdown  of  the  errors  by  type. 
From  this,  we  can  see  that  the  vast  majority  (79*)  of  the  errors  were 
due  to  incorrect  code.  Most  of  the  remaining  errors  were  attributable 
to  incorrect  design. 

The  activities  used  in  an  attempt  to  detect  errors  were  mostly  com¬ 
pilation,  design  reading,  design  walkthroughs,  code  reading,  or  some 
combination  of  these.  Approximately  half  of  the  errors  were  success¬ 
fully  detected  through  compiler  messages,  and  a  slightly  smaller  number 
were  successfully  detected  through  readings  and  walkthroughs.  These 
same  activities  were  used  to  isolate  the  source  of  the  error.  Code 
reading  was  more  successful  at  isolating  the  source  of  the  error  than  at 
detecting,  it,  and  the  opposite  is  true  of  compiler  messages.  In  the 
case  of  design  reading  and  walkthroughs,  detection  of  the  error  and  iso¬ 
lation  of  its  source  usually  took  place  simultaneously,  but  in  many 
cases  the  programmer  only  checked  the  detection  columns  on  the  form.  A 
complete  listing  of  the  activities  used  to  detect  and  Isolate  errors  is 
provided  in  Appendix  4. 


Table  4.8  -  Breakdown  of  Errors  by  Type 


Type  of  Error 

Number 
of  Errors 

Percentage 

requirements  incorrect 

2 

1.04* 

requirements  misinterpreted 

4 

2.08* 

design  incorrect 

30 

15.63* 

design  misinterpreted 

0 

0.00* 

code  incorrect 

151 

78.65* 

external  environment 

0 

0.00* 

misunderstood  (not  language 

or  compiler) 

clerical  error 

5 

2.60* 

Over  Son  of  the  errors  took  at  most  twelve  minutes  (0.2  hour)  to 
isolate.  Approximately  as  many  errors  required  as  little  time  to 
correct.  Only  seven  errors  took  an  hour  or  more  either  to  isolate  or  to 
correct.  One  error  took  an  hour  to  isolate  but  only  required  0.1  hour 
to  correct.  It  was  a  design  incorrect  error  which  involved  renaming  a 
file.  Another  error  classified  as  code  incorrect  took  two  hours  to  iso¬ 
late  but  only  0.3  hour  to  correct.  An  undefined  part  of  a  string  was 
passed  as  an  argument  to  a  function.  Two  errors  involving  incorrect 
design  each  required  only  0.1  hour  to  isolate  but  over  an  hour  to 
correct.  One  of  these  was  a  tasking  error  involving  a  synchronization 
problem  between  two  components  and  it  required  5.2  hours.  Another, 
which  required  1.5  hours,  was  a  logic  error  involving  input/output.  The 
remaining  three  errors  took  an  hour  or  more  to  isolate  and  an  hour  or 
more  to  correct.  Two  of  them  involved  incorrect  code.  One  required  the 
insertion  of  error  checks  and  exception  handlers  in  a  routine  to  conform 
to  the  specifications;  this  took  one  hour  to  isolate  and  one  hour  to 
correct.  The  other  took  four  hours  to  isolate  and  four  hours  to 
correct;  it  was  an  input/output  syntax  error.  The  last  error  which  took 
one  hour  to  isolate  and  one  hour  to  correct  was  a  requirements  incorrect 
error.  A  superfluous  requirements  section  was  found,  and  this  was  even¬ 
tually  deleted.  Table  4.9  shows  the  time  required  to  isolate  errors; 
Table  4.10  gives  a  breakdown  by  error  type  of  effort  needed  to  correct 
errors . 

Table  4.11  shows  when  errors  entered  the  system.  Seventy-two  per¬ 
cent  occurred  during  the  Ada  coding  stage.  Twenty- four  percent  also 
entered  the  system  during  design.  As  stated  previously,  several  of  the 
errors  reported  as  coding  errors  actually  originated  in  the  design 
stage . 

A  summary  of  the  different  types  of  errors  and  the  highest  level 
document  that  had  to  be  changed  for  each  one  is  presented  in  Table 
4.12a.  Eight  errors  where  the  code  was  incorrect  involved  changes  in 
the  PDL.  This  may  seem  anomalous.  However,  these  errors  resulted  in  mere 
code  changes  to  the  PDL  document  and  not  changes  to  the  design  itself. 

Table  4.9  -  Time  to  Isolate  Errors 


Effort  (hours) 

Number  of  Errors 

Percentage 

0.1 

115 

59.90 

0.2 

44 

22.92 

0.3 

10 

5.21 

0.5 

9 

4.69 

0.6 

2 

1.04 

0.8 

5 

2.60 

0.9 

2 

1.04 

1.0 

3 

1.56 

2.0 

1 

0.52 

4.0 

1 

0.52 

Mean  =  0.23  hours  per  error 
Standard  Deviation  =  0.36  hours  per  error 
Median  =  0.10  hours  per  error 


Table  H.10  -  Time  to  Correct  Errors 


Effort  (hours) 

Number  of  Errors 

Percentage 

0.1 

116 

503 

0.2 

47 

24.48 

0.3 

12 

6.25 

0.4 

5 

2.60 

0.5 

5 

2.60 

0.6 

2 

1.04 

1.0 

2 

1.04 

1.5 

1 

0.52 

4.0 

1 

0.52 

5.2 

1 

0.52 

Mean  =  0.22  hours  per  error 
Standard  Deviation  «  0.48  hours  per  error 
Median  =  0.10  hours  per  error 

Table  4.11  -  Stage  in  Vfhieh  Error  Entered  the  System 


Stage  of  the  Project 

Number  of 
Errors 

Percentage 

requirements 

2 

1.04 

design 

46 

23.96 

Ada  coding 

139 

72.40 

testing 

1 

0.52 

implementing  another  change 

4 

2.08 

other  or  can't  tell 

0 

0.00 

(For  exeunple,  a  semicolon  error  in  the  PDL  document  would  have  been 
counted  as  incorrect  code,  not  incorrect  design.)  Table  4.12b  shows  the 
number  of  errors  per  thousand  lines  of  code. 

The  number  of  interface  errors  (those  errors  which  entailed  modifi¬ 
cations  in  more  than  one  component  at  the  same  level  of  document)  was 

Table  4.12a  -  Type  of  Error  vs.  Highest  Level  Document  Changed 


Type  of  Error 

Number  of  Errors 

Req. 

PDL 

code 

module 

requirements  incorrect 

2 

0 

0 

requirements  misinterpreted 

0 

4 

0 

design  Incorrect 

0 

30 

0 

design  misinterpreted 

0 

0 

0 

code  incorrect 

0 

8 

143 

external  environment 

0 

0 

0 

misunderstood 

clerical  error 

0 

0 

5 

TOTAL 

2 

42 

"Tli5 

PERCENTAGE 

1.04 

21.88 

77. 06 

Table  4. 12b  -  Number  of  Errors  per  Thousand  Lines  of  Code 


Errors  in  Code 

All  Errors 

Modules  Only 

Text 

31 

41 

Non-blank  lines 

17 

22 

calculated.  Only  1011  of  all  the  errors  were  interface  errors.  Approxi¬ 
mately  half  were  in  the  design,  and  half  were  in  the  code.  The  maximum 
number  of  components  involved  for  zuiy  single  error  was  three.  (See 
Table  4.13.) 

Appendix  5  contains  the  distribution  of  errors  over  the  software 
development  cycle.  The  errors  displayed  a  normal  development  pattern. 

As  explained  in  Section  3«3»  the  errors  were  classified  as  follows: 

a)  Language 

b)  Problem 

c)  Clerical 

Language  errors  were  those  which  involved  the  syntax  or  semantics  of  a 
feature  or  those  which  involved  the  concept  behind  a  feature.  The  prob¬ 
lem  category  encompassed  logic  errors  and  errors  related  to  the  environ¬ 
ment.  Clerical  errors  Included  those  due  to  carelessness,  e.g.  typo¬ 
graphical  errors. 

Of  the  192  error  description  forms  examined,  146  (.76%)  claimed  that 
the  use  of  Ada  contributed  to  the  error.  As  shown  in  Table  4.14,  the 
vast  majority  of  the  errors  were  language  errors,  and  furthermore,  6911 
of  these  were  merely  syntax  errors,  which  explains  why  so  many  of  the 
errors  took  so  little  time  to  correct.  There  were  2*»  syntax  errors  per 
thousand  lines  of  text  (any  line  containing  part  of  an  Ada  statement) 
and  13  syntax  errors  per  thousand  non-blank  lines.  The 
language/problem/clerical  classification  is  fu»*ther  broken  down  by  error 
type  in  Table  4.15.  As  might  be  expected,  most  of  the  errors  involving 
requirements  were  problem  errors,  and  most  of  the  errors  involving 
incorrect  design  or  code  were  language-related  errors. 


Table  4.13  -  Interface  Errors 


Document 

Number  of  Errors 

2  components 

3  components 

involved 

involved 

Requirements 

0 

0 

PDL 

5 

4 

Code 

10 

0 

15 


Table  4.14  -  Number  of  Language,  Problem  and  Clerical  Errors 


Language 

166 

Concept 

7 

Syntax 

114 

Semantics 

45 

Problem 

21 

Clerical 

5 

Table  4.15  -  Type  of  Error  vs.  Language,  Problem  or  Clerical  Classifica¬ 
tion 


Type  of  Error 

Number  of  Errors 

Language 

Problem 

Clerical 

requirements  incorrect 

0 

2 

0 

requirements  misinterpreted 

1 

3 

0 

design  incorrect 

25 

5 

0 

design  misinterpreted 

0 

0 

0 

code  incorrect 

140 

11 

0 

external  environment 

0 

0 

0 

misunderstood 

clerical  error 

0 

0 

5 

TOTAL 

166 

21 

5 

Several  Ada  language  features  were  involved  in  errors.  Most  common 
among  these  were  low-level  syntax  (e.g.  semicolon,  parenthesis,  assign¬ 
ment)  and  loops.  There  were  also  a  considerable  number  of  errors 
involving  tasks,  separate  compilation,  generics,  procedures/functions, 
parameters,  and  declarations.  As  previously  stated,  most  of  the  errors 
were  syntax  errors.  There  were  only  seven  concept  errors,  and  these 
Involved  tasking,  exceptions,  access  types,  file  Input/output  and  pack¬ 
ages.  Of  the  forty- five  semantic  errors,  there  were  six  each  involving 
parameters  and  generics  and  five  each  involving  compilation  units  and 
declarations.  (See  Table  4.16a.) 

The  errors  involving  the  various  Ada  language  features  were  normal¬ 
ized  with  respect  to  the  number  of  times  each  feature  was  used.  (See 
Table  4.16b.)  For  example,  there  were  eight  errors  involving  tasks  and 
21  occurrences  of  tasking  in  the  project;  this  gives  a  38>  error  rate 
for  tasking  which  was  the  highest  percentage  for  any  feature.  Access 
types  and  generics  had  error  rates  of  27%  and  24%  respectively.  Compi¬ 
lation  units  and  PRAGMA  each  had  a  13%  error  rate.  It  is  worthwhile  to 
note  that  all  of  these  features  are  specific  to  Ada. 

The  error  description  forms  included  questions  to  assess  the  pro¬ 
grammers'  comprehension  of  Ada  features.  For  errors  in  the  PDL  or  code, 
the  programmer  said  that  the  documentation  explained  the  features 
clearly  in  most  cases  and  that  he  understood  the  features  but  did  not 
apply  them  correctly.  In  some  cases,  the  programmer  did  not  understand 
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Table  4.16b  -  Errors  Normalized  with  Respect  to  Feature  Usage 


Ada  Language 
Feature 

Concept 

Percentage  o] 
Semantics 

f  Errors 
Syntax 

TOTAL 

semicolon 

0 

0 

0.48 

0.48 

parenthesis 

0 

0 

3.81 

3.81 

colon 

0 

0 

0.28 

0.28 

•  « 

•  » 

0 

0 

0.82 

0.82 

comment 

0 

0 

0.08 

0.08 

Identifier 

0 

0.27 

1.07 

1.34 

loop 

0 

0 

6.55 

6.55 

CASE 

0 

0 

5.88 

5.88 

IF 

0 

0 

2.73 

2.73 

BEGIN/END 

0 

0 

2.08 

2.08 

RETURN 

0 

0 

0.83 

0.83 

aggregate 

0 

0.06 

0 

0.06 

strings 

0 

0.38 

0 

0.38 

arrays 

0 

0.21 

0.14 

0.35 

records 

0 

0.25 

0.25 

0.50 

access  type 

9.09 

18.18 

0 

27^7 

declarations 

0 

0.37 

0.59 

0.96 

parameters 

0 

0.96 

0.63 

1.59 

procedures/ 

0 

0.70 

1.23 

1.93 

functions 

tasking 

9.52 

9.52 

19.05 

38.09 

exceptions 

1.01 

0 

0.51 

1.52 

generics 

0 

18.18 

6.06 

24.24 

pacluiges 

6.25 

0 

6.25 

12.50 

compilation  units 

0 

10.00 

4.00 

14.00 

attributes 

0 

0.70 

0 

0.70 

PRAGMA 

0 

12.50 

0 

12.50 

the  features  fully,  and  in  a  few  others  he  understood  the  features 
separately  but  not  their  interactions.  The  programmer  usually  remem¬ 
bered  how  the  features  should  be  applied  or  obtained  information  from 
another  programmer  to  correct  the  error.  For  more  details,  see  Appendix 

6. 


4.2.4  Prograunmers 


Table  4.17a  shows  a  breakdown  of  the  effort  for  each  programmer. 

The  productivity  of  each  programmer  is  shown  in  Table  4.17b.  The  types 
of  changes  versus  the  programmer  who  authored  the  document  in  which  the 
change  was  made  are  presented  in  Table  4.18a,  while  the  types  of  changes 
requested  by  each  programmer  are  presented  in  Table  4.18b.  In  all 
except  67  cases,  the  programmer  who  requested  the  change  was  the  same 
person  who  had  authored  the  document  in  which  the  change  was  made.  In 
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Tables  4.19a  and  4.19b,  we  see  how  many  of  each  type  of  error  the  vari¬ 
ous  programmers  committed  and  found.  Table  4.19c  shows  the  number  of 
errors  per  thousand  lines  of  code  for  each  programmer. 

The  lead  programmer  who  had  the  most  industrial  experience  and  the 
most  experience  in  the  specific  application  area  spent  most  of  his  time 
working  on  the  requirements  of  the  project,  and  a  considerable  amount  of 
time  on  design.  He  made  and  found  the  vast  majority  of  the  design 
errors . 

The  senior  programmer  worked  mostly  on  design.  He  made  and  found 
all  four  of  the  requirements  misinterpreted  errors.  Note  that  the 
highest  level  document  changed  for  these  errors  was  the  PDL.  He  did  not 
do  much  coding,  but  was  by  far  the  moat  productive  of  the  four.  He 
averaged  39.85  lines  of  text  per  hour  during  code  development  (six  times 
more  productive  than  the  other  programmers)  and  2.8l  lines  of  text  per 
hour  for  the  entire  project. 

The  junior  programmer  spent  almost  an  equal  amount  of  time  on 
design  and  coding.  He  seemed  to  have  the  easiest  time  grasping  ideas  in 
Ada,  but  he  had  the  highest  error  rate.  He  did  most  of  the  unit  testing 
and  therefore  found  most  of  the  coding  errors.  He  and  the  senior  pro¬ 
grammer  requested  80K  of  the  changes.  They  also  made  the  most  coding 
errors,  but  they  had  written  the  most  code  and  their  code  was  tested 


Table  4.17b  -  Productivity  of  Each  Programmer 


Productivity 

(Lines  of  Code  per  Hour) 

Code  Developed 

Amount  of 

Entire  Project 

Code  Developed 

Text 

Lead 

6.56 

0.58 

Senior 

39.85 

2.81 

Junior 

6.08 

0.83 

Librarian 

6.33 

0.23 

All  Non-blank 

Lead 

11.50 

1.02 

Lines 

Senior 

71.44 

5.04 

Junior 

18.47 

2.52 

Librarian 

7.78 

0.28 

Note  -  Text  refers  to  any  line  containing  part  of  an  Ada  statement. 


Table  4.18a  -  Types  of  Changes  vs.  Document  Author 


Type  of  Change 

Lead 

Senior 

Junior 

Librarian 

Total  Changes 

4l” 

143 

139 

7 

and  %  of  all  changes 

14.24 

42.43 

41.25 

2.08 

Requirements 

9 

4 

10 

1 

PDL 

29 

45 

33 

0 

Code  Module 

10 

94 

96 

6 

Error  Corrections 

28 

75 

85 

4 

Changes  in  Problem  Domain 

1 

0 

0 

0 

Planned  Enhancements 

3 

3 

3 

0 

Avoidances  of  Apparent 

Problems  with  the  Compiler 

0 

7 

11 

0 

Avoidances  of  Other  Problems 
in  the  Developing  Env. 

0 

1 

1 

0 

Adaptations  to  a  Change  in 
the  Developing  Env. 

0 

6 

0 

1 

Improvements  of  Clarity, 

Malnt.  or  Documentation 

9 

37 

28 

2 

Optimization  of  Time,  Space 
or  Accuracy 

0 

1 

1 

0 

Insertion  or  Deletion  of 

Debug  Code 

0 

7 

2 

0 

Other  Than  Above 

7 

6 

8 

0 

20 


Table  4.18b  -  Types  of  Changes  Requested  by  Each  Programmer 


Type  of  Change 

Lead 

Senior 

Junior 

Librarian 

Total  Changes 

56 

90 

178 

13 

and  %  of  all  changes 

16.62 

26.70 

52.81 

3.86 

Requirements 

9 

4 

10 

1 

PDL 

29 

43 

34 

1 

Code  Module 

18 

43 

134 

11 

Error  Corrections 

31 

49 

104 

8 

Changes  in  Problem  Domain 

1 

0 

0 

0 

Planned  Enhancements 

Avoidances  of  Apparent 

3 

1 

3 

2 

Problems  with  the  Compiler 
Avoidances  of  Other  Problems 

0 

4 

14 

0 

in  the  Developing  Env. 
Adaptations  to  a  Change  in 

0 

1 

1 

0 

the  Developing  Env. 
Improvements  of  Clarity, 

0 

2 

4 

1 

Malnt.  or  Documentation 
Optimization  of  Time,  Space 

9 

32 

33 

2 

or  Accuracy 

Insertion  or  Deletion  of 

0 

1 

1 

0 

Debug  Code 

0 

0 

9 

0 

Other  Than  Above 

12 

0 

9 

0 

Table  4.19a  -  Types  of  Errors  Made  by  Each  Programmer 


Type  of  Error 

Number  of  Errors 

Lead 

Senior 

Junior 

Librarian 

requirements  incorrect 

1 

0 

1 

0 

requirements  mis- 

0 

4 

0 

0 

Interpreted 

design  incorrect 

17 

9 

4 

0 

design  misinterpreted 

0 

0 

0 

0 

code  incorrect 

10 

60 

77 

4 

external  environment 

0 

0 

0 

0 

misunderstood 

clerical  error 

0 

2 

3 

0 

21 


eo  o  <vi 


1 


Table  H.igb  -  Types  of  Errors  Found  by  Each  Programmer 


Type  of  Error 

Number  of  Errors 

Lead 

Senior 

Junior 

Librarian 

requirements  incorrect 

1 

0 

1 

0 

requirements  mis- 

0 

4 

0 

0 

interpreted 

design  incorrect 

17 

9 

4 

0 

design  misinterpreted 

0 

0 

0 

0 

code  incorrect 

13 

33 

97 

8 

external  environment 

0 

0 

0 

0 

misunderstood 

clerical  error 

0 

3 

2 

0 

Table  4.19c  -  Error  Rate  (Errors  per  thousand  lines  of  code) 


Type  of  Code 

Num 

Lead 

ber  of  Er 
Senior 

rors  per 
Junior 

1000  LOG 
Librarian 

Text 

Non-blank  lines 

38.6 

22.0 

34.4 

19.2 

51.1 

24.1 

25.5 

20.7 

more  fully  than  the  other  programmers'  code.  In  addition,  they  imple¬ 
mented  most  of  the  improvements  of  clarity,  maintainability  and  documen¬ 
tation. 

The  librarian  who  had  the  least  programming  experience  spent  the 
least  time  on  coding;  he  wrote  only  one  module,  1.74JI  of  the  total 
amount  of  non-blank  lines  of  code.  He  was  mainly  responsible  for 
librarian  duties.  He  requested  only  4%  of  the  changes,  and  these  were 
mostly  error  corrections.  He  made  four  coding  errors  but  caught  a  total 
of  eight  coding  errors. 

A  breakdown  by  author  of  errors  classified  as  language,  problem  and 
clerical  is  presented  in  Table  4.20a.  The  senior  programmer  was  respon¬ 
sible  for  67/1  of  the  problem  errors,  probably  because  he  worked  mostly 
on  design.  He  and  the  junior  programmer  made  835J  of  the  language 
errors.  Furthermore,  the  Junior  programmer  made  over  half  of  all  the 
syntax  errors  and  had  the  highest  rate  for  these  errors  as  shown  in 
Table  4.20b.  However,  it  should  be  noted  that  the  Junior  programmer 
probably  tested  his  own  code  more  thoroughly  than  the  other  programmers' 
code  using  the  ROLM  compiler. 

Table  4.21a  shows  the  errors  categorized  by  Ada  language  feature 
for  each  programmer  and  Table  4.21b  shows  the  errors  involving  each 
feature  normalized  by  usage  for  each  programmer.  Although  tasking,  gen¬ 
erics  and  compilation  units  were  not  used  much,  they  presented  the  most 
problems  to  all  the  programmers  who  used  them.  Again,  it  should  be 
emphasized  that  these  features  are  unique  to  Ada.  The  lead  programmer 
had  the  highest  error  rate  for  these  three  features  but  he  did  not  use 
them  as  much  as  the  Junior  and  senior  programmers  did.  Exceptions  were 


m 


> 


22 


Table  4.20a  -  Error  Classification  by  Programmer 


Error  Class 

Number  of  Errors 

Lead 

Senior 

Junior 

Librarian 

TOTAL 

language 

25 

59 

78 

4 

166 

concept 

2 

5 

0 

0 

7 

syntax 

17 

34 

60 

3 

114 

semantics 

6 

20 

18 

1 

45 

problem 

3 

14 

4 

0 

21 

clerical 

0 

2 

3 

0 

5 

Table  4.20b  -  Syntax  Errors  per  Thousand  Lines  of  Code  for  Each  Program¬ 
mer 


Type  of  Code 

Number 

of  Syntax  Errors  per  1000  LOC  I 

Lead 

Senior 

Junior 

Librarian 

Text 

23.4 

15.6 

36.1 

19.1 

Non-blank  lines 

13.4 

8.7 

17.0 

15.5 

used  a  total  of  117  times  and  only  the  lead  programmer  had  difficulty 
using  this  feature.  Two  of  the  three  exception  errors  he  made  were  con¬ 
cept  errors.  Eighty-eight  percent  of  the  PRAGMA  usage  was  by  the  lead 
programmer  and  both  of  the  PRAGMA  errors  were  made  by  him.  Six  out  of 
the  eleven  uses  of  access  types  were  by  the  senior  programmer;  all  three 
of  the  errors  were  made  by  him.  Being  a  recent  computer  science  gradu¬ 
ate,  the  junior  programmer  was  most  fauniliar  with  Pascal.  It  is 
interesting  to  note  that  he  was  not  responsible  for  any  of  the  concept 
errors. 

Table  4.22  shows  how  the  programmers  responded  to  questions  on  the 
error  description  forms  regarding  understanding  Ada  features.  As 
expected,  the  senior  programmer  and  the  Junior  programmer  who  were  the 
most  well-versed  in  high-level  languages  understood  features  in  Ada  and 
tried  to  apply  them,  sometimes  unsuccessfully  as  evidenced  by  the 
errors.  However,  they  recognized  them  readily  and  tried  to  correct 
them.  The  lead  programmer  had  the  moat  difficulty  understanding  Ada 
features. 


Table  4.21b  -  Errors  Normalized  with  Respect  to  Feature  Usage  for  Each 
Programmer 


Ada  Language 

Percentage  of  Errors 

Feature 

Lead 

Senior 

Junior 

Librarian 

semicolon 

0 

0.20 

0.97 

0.03 

parenthesis 

9.3 

4.10 

2.65 

0 

colon 

2.74 

0.13 

0 

0 

•  « 

•  • 

0 

0.81 

1.29 

0 

comment 

0 

0.10 

0.09 

0 

identifier 

0.76 

0.67 

4.23 

0 

loop 

8.00 

3.77 

9.26 

0 

CASE 

3.33 

0 

0 

« 

IF 

0 

3.13 

3.13 

6.25 

BEGIN/END 

0 

1.64 

3.26 

0 

RETURN 

0 

0 

1.64 

0 

aggregate 

0 

0.14 

0 

0 

strings 

0 

0.68 

0 

« 

arrays 

0 

0.48 

0.41 

0 

records 

0 

0.36 

0.88 

access  type 

0 

50.00 

0 

• 

declarations 

0 

0.54 

1.70 

3.22 

parameters 

6.25 

1.01 

1.60 

8.33 

procedures/ 

4.55 

0.89 

3.50 

0 

functions 

tasking 

100.00 

38.46 

28.57 

* 

exceptions 

11.11 

0 

0 

0 

generics 

50.00 

11.76 

35.71 

« 

packages 

» 

20.00 

9.10 

• 

compilation  units 

50.00 

15.00 

8.00 

0 

attributes 

0 

1.85 

0 

0 

PRAGMA 

14.29 

« 

0 

» 

Note  -  An  means  that  this  programmer  never  used  this  feature 


Table  4.22  -  Understanding  Features  By  Programmer 


UNDERSTANDING  FEATURES  BY  PROGRAMMER 


Lead 

Senior 

Junior 

no  reply 

3 

3 

2 

understood  features 
separately,  but  not 
their  interaction 

1 

1 

6 

understood  features 
but  did  not  apply 
them  correctly 

15 

44 

83 

did  not  understand 
features  fully 

12 

1 

12 

confused  feature  with 
a  feature  in  another 
language 

0 

0 

1 

1 


5.  SUMMARY  AND  CONCLUSIONS 


This  report  analyzes  the  data  from  the  development  of  a  system  in 
Ada  by  four  programmers  at  GE.  The  data  analyzed  include  effort,  change 
and  error  data  as  well  as  basic  metrics  on  the  size  of  the  project  and 
features  of  the  language  used.  The  project  was  not  completed,  and  lit¬ 
tle  time  was  spent  on  testing.  (Only  unit  testing  was  done  and  even 
that  was  not  completed.)  Several  modules  were  never  coded. 

The  majority  of  the  changes  were  error  corrections.  Most  of  the 
remaining  changes  were  improvements  of  clarity,  maintainability  or  docu¬ 
mentation.  The  highest  level  document  changed  was  the  code  module  in 
most  cases.  The  vast  majority  of  the  coding  errors  were  code  incorrect. 
In  addition,  design  Incorrect  accounted  for  16$.  There  are  probably 
many  more  errors  still  in  the  system  since  it  was  not  fully  tested. 

A  large  number  of  language  errors  were  made.  Many  Involved  Ada- 
specific  features.  The  programmers  used  most  of  the  language  features 
of  Ada,  but  not  necessarily  as  they  were  intended  by  the  language 
designers.  The  error  rates  of  the  Ada-specific  features  were  generally 
very  high.  There  were  only  a  few  concept  errors,  and  these  involved 
tasking,  exceptions,  access  types,  packages  and  file  Input/output. 
Tasking,  generics  and  compilation  units  were  not  used  much,  but  they 
presented  the  most  problems  to  all  the  programmers  who  used  them.  These 
features  are  unique  to  Ada. 

The  need  for  change  was  determined  in  less  than  an  hour  for  almost 
all  of  the  changes.  In  addition,  the  time  to  design  and  Implement  the 
change  was  one  hour  or  less  for  almost  all  of  the  changes.  Most  errors 
took  less  than  fifteen  minutes  to  Isolate  and  as  little  time  to  correct. 
(Many  of  the  errors  were  syntax  errors.) 

Because  of  the  learning  curve,  it  was  not  possible  to  Judge  the 
impact  of  Ada  on  schedules,  costs  or  milestones.  Because  Ada  is  a  com¬ 
pletely  new  language  with  features  not  present  in  other  programming 
languages,  about  20$  of  the  total  effort  was  spent  on  training  and 
methodology,  which  is  more  than  the  effort  on  any  other  phase  of  the 
project.  This  is  a  much  higher  percentage  than  would  typically  be  spent 
on  most  projects.  Furthermore,  even  this  amount  seems  insufficient 
because  the  programmers  indicated  that  they  did  not  feel  comfortable 
with  Ada  until  after  they  left  the  project. 

Lack  of  support  tools  discouraged  the  programmers.  This  shows  that 
automated  tools  are  paramount  for  success  in  a  software  project.  This 
is  especially  true  given  a  language  with  the  complexity  of  Ada.  Many 
syntax  errors  were  uncovered,  and  prograunmers  could  have  spent  this  time 
looking  for  logic  errors.  Tools  needed  include  a  structured  editor, 
data  dictionaries,  call  structure  and  compilation  dependency  tools,  and 
cross  references.  Errors  could  be  caught  earlier  in  development  with  a 
PDL  processor.  (The  earlier  an  error  is  caught,  the  less  expensive  it 
is  to  fix  in  general.) 

The  greatest  error  rate  appeared  to  be  associated  with  the  moat  Ada 
specific  features:  tasking,  generics  and  compilation  units.  The  lead 


prograunmer  had  the  highest  error  rate  in  these  categories.  The  Junior 
programmer  did  not  make  any  of  the  concept  errors  and  he  seemed  to  have 
the  easiest  time  grasping  Ideas  In  Ada.  The  Jjnlor  progrsuomer  had 
recently  graduated,  while  the  lead  programmer  had  worked  In  industry  for 
many  years. 

There  Is  further  analysis  that  can  and  will  be  done.  The  effec¬ 
tiveness  of  the  features  will  be  examined.  The  use  of  packages  has 
already  been  studied  [Gannon,  et  al.  83]. 


Appendix  2  -  Improvements  of  Clarity,  Maintainability  or  Documentation 

by  Type  of  Document 


Type  of  Document  I 

Number  of 

Changes  1 

1 

Percentage 

Requirements  I 

14 

1 

1 

18. 42$ 

PDL  1 

54 

1 

71.05$ 

Code  Module  1 

8 

1 

10.53$ 

Total  1 

76 

1 

100.00$ 

changes 

made 


Appendix  4  -  ACTIVITIES  USED  TO  DETECT  AND  ISOLATE  ERRORS 


1 

DETECTING  ERROR:  1 

ISOLATING 

SOURCE: 

ACTIVITIES! 

ACTIVITIES  1 

ACTIVITIES! 

ACTIVITIES 

USED  FOR  1 

SUCCESSFUL  1 

TRIED  TO  1 

SUCCESSFUL 

PROGRAM  i 

IN  DETECTING  1 

FIND  CAUSE! 

IN  FINDING 

validation! 

! 

ERROR  SYMPTOMS! 

1 

1 

! 

CAUSE 

design  reading 

! 

34  ! 

1 

24  1 

1 

19  1 

17 

design  walkthrough 

35  ! 

31  I 

18  1 

15 

code  reading 

58  ! 

31  1 

57  1 

57 

code  walkthrough 

2  ! 

1  1 

3  ! 

2 

talk  with  other  programmer 

2  ! 

2  ! 

2  ! 

2 

reading  documentation 

3  ! 

1  1 

6  ! 

4 

compiler  messages 

105  ! 

105  ! 

76  ! 

75 

system  error  messages 

4  ! 

4  ! 

3  ! 

3 

project  error  messages 

1  ! 

1  ! 

1  1 

1 

trace 

5  ! 

5  1 

6  ! 

6 

dump 

0  ! 

0  ! 

0  ! 

0 

inspection  of  output 

1  ! 

1  ! 

1  1 

0 

pre-acceptance  test  run 

4  ! 

2  1 

1  1 

1 

acceptance  test 

0  ! 

0  ! 

0  1 

0 

Ada  runtime  check 

0  ! 

0  1 

0  ! 

0 

other 

4  ! 

! 

4  1 

! 

2  1 

1 

2 

Appendix  6  -  FOR  AN  ERROR  IN  THE  PDL  OR  CODE 


DOES  THE  DOCUMENTATION  EXPLAIN  THE  FEATURE  CLEARLY? 


not  applicable 

16 

8.33$ 

yes 

157 

81.77$ 

no 

19 

9.90$ 

WHICH  OF  THE  FOLLOWING  IS  MOST  TRUE? 


no  reply 

8 

4.17$ 

understood  features 

separately, 

but  not  their  interaction 

9 

4.69$ 

understood  features 

but  did  not 

apply  them 

correctly 

147 

76.56$ 

did  not  understand 

features  fully 

27 

14.06$ 

confused  feature  with  a  feature 

in  another 

language 

1 

0.52$ 

WHERE  THE  INFORMATION  NEEDED  TO  CORRECT  ERROR  WAS  FOUND 


no  place  specified  4 
class  notes  6 
Ada  reference  manual  7 
another  programmer  19 
remembered  144 
viewgraphs  from  tapes  0 
test  program  1 
other  16 


Appendix  7 • 1 

CHANGE  REQUEST  * _ 

^e’-son  requesting  change  _ Approved  by  _ ^Date  _ / _ / 

1.  V/hat  is  the  reason  for  this  change?  _ 

2.  When  was  the  need  for  the  change  determined?  /  / _ 

3.  Describe  the  necessary  change:  _ 


4.  To  completely  implement  the  desired  change,  the  highest  level  document  that 
needs  changing  is: 

(complete  one:)  requirements  section;  _ 

PDL  module:  _ 

code  module: 


5.  When  did  the  effort  begin  to  understand  and  isolate  this  change?  _ / _ / _ 

6.  How  much  effort  has  been  spent  so  far  in  isolating  and  understanding 
what  needs  to  be  changed? 

l  I  I  1  !  \  \ 

1  hr.  4  hrs.  2  days  1  wk.  2  wks.  1  mo.  2  mos. 

7.  This  change  is  a/an  (check  one): 

error  correction  (attach  completed  ERROR  DESCRIPTION  FORM) 

_  change  in  the  problem  oomain 

_  planned  enhancement 

_  avoidance  of  an  apparent  problem  with  the  compiler  (explain  below) 

_ avoidance  of  some  other  problem  in  the  development  environment 

(explain  below) 

_ adaptation  to  a  change  in  the  development  environment 

(describe  below) 

_ improvement  of  clarity,  maintainability  or  documentation 

_ optimization  of  time,  space  or  accuracy 

_  insertion  or  deletion  of  debug  code 

_ other  than  above  (describe  below) 

(over) 


^  or  2 


List  all  known  documents  which  will  require  a  change  as  a  result  of  this 
change  ’-equest  to  maintain  system  consistency: 

(to  be  filled  in  by  lioreri 
date  change 

Rqmts.  POL  Code  ,  section  nos. /module  name  submitted  to  librarian 

‘l _ ! _ ! _ !  _  _ / _ / _ 

! _ ! _ ! _ !  _  _ ! _ ! _ 

! _ ! _ ! _ !  _  _ ! _ t _ 

! _ ! _ ! _ !  _  _ ! _ / _ 

! _ ! _ ! _ !  _  _ ! _ / _ 

! _ ! _ ! _ !  _  _ ! _ ! _ 

! _ ! _ ! _ !  _  _ ! _ t _ 

1 _ ! _ ! _ !  _  _ / _ J _ 

i _ ! _ ! _ !  _  _ / _ / _ 

! _ ! _ ! _ !  _  _ ! _ / _ 

! _ ! _ ! _ !  _  _ / _ ! _ 

! _ ! _ ! _ !  _  _ _ / _ 

! _ ! _ ! _ !  _  _ ! _ / _ 

I _ ! _ 1 _ I  _  _ ! _ / _ 

! _ ! _ ! _ !  _  _ ! _ ! _ 

!  !  !  !  II 


(Note  that  each  of  these  will  require  a  separate  Individual  Document  Change  Report.) 


9.  What  additional  documents  were  examined  or  will  be  examined  in  determining  the  chai 


Rqmts.  POL  Code 


section  nos. /module  name 


X  a^c  X  u  X  4^ 


Appendix  7 . 2 


ERROR  DESCRIPTION  FORM  for  CHANGE  REQUEST  # 


Type  of  Error: 

_  requirements  incorrect 

_  requirements  misinterpreted 

_  *design  incorrect 

_  ’design  misinterpreted 

_  ’code  incorrect 

_ external  environment  misunderstood  (not  language  or  compiler) 

_  clerical  error 

’Was  the  error  in  the  use  of  data  or  in  function  _  ? 


Did  the  use  of  Ada  as  a  design  and  implementation  language  contribute  to  this 
error?  _  If  so,  was  it  only  a  syntax  error?  _ 


Whether  related  to  Ada  or  not,  which  language  features  were  involved  in  tne  error? 


A.  For  an  error  in  the  POL  or  code: 

a.  does  the  documentation  explain  the  feature  clearly? _ Yes _ No 

b.  which  of  the  following  is  most  true? 

_  understood  features  separately,  but  not  their  interaction 

_  understood  features  but  didn't  apply  them  correctly 

_  didn't  understand  features  fully 

_  confused  feature  with  a  feature  in  another  language 

c.  where  was  the  information  needed  to  correct  the  error  found? 

_  class  notes 

_  Ada  reference  manual 

_  another  programmer 

_  remembered 

_  viewgraphs  from  tapes 

_  test  program 

other: 
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Detecting  error;  1 

1 

Activities 
Used  for 
Program 

Val idation 

Activities 
Successful 
in  Detecting 
Error  Symptoms 

Oesian  reading 

Design  waT< through 

Code  reading 

Code  walkthrough 

Talk  w/other  programmer 

Reading  documentation 

Compiler  messages 

System  error  messages 

Project  error  messages 

Trace 

Dump 

Inspection  of  output 

Pre-acceptance  test  run 

Acceptance  test 

Ada  runtime  checking 

Other: 

. 

Isolating  source: 

Activities  Activities 

Tried  to  Successful 

Find  Cause  in  Finding 

Cause 

6.  What  was  the  tine  used  to  isolate  the  source  of  the  error? 

I  I  t  t  I  I  I 

_ • _  _ 4 _  _ 4  _ _ • _ • _ >  * 

1  hr.  4  hrs.  2  days  1  wk.  2  wks.  1  mo.  2  mos. 

If  never  found,  was  a  workaround  used?  _  (Explain  in  8,  below.) 

7.  When  did  the  error  enter  the  system? 

_ requirements  _ design  _ Ada  coding  _ testing 

_  implementing  another  change.  Change  Report  No.  _ 

_  other  or  can't  tell  (explain  below) 


a.  Use  this  space  to  give  any  adoitional  information  that  might  help  in  understanaing 
the  cause  of  the  change  and  its  ramifications: 


Appendix  7.3 

INDIVIDUAL  DOCUMENT  CHANGE  REPORT 

1.  This  change  has  been  approved  through  CHANGE  REQUEST  # _ 

2.  Document  being  changed  (complete  one): 

requirements  spec,  section  _ 

POL  module:  _ 

code  module: 


3,  How  much  effort  (person-time)  was  spent  changing  this  document  (not  librarian's  time)? 

I  f  I  I  I  I  I  I 

_ _ •  *  • _ • _ • _ •  •  • 

T3mTn  Thr  4hrs  Tday  2d  ays  Twic  2v7k  Inio 

4.  Person  responsible  for  this  change _ Date  _ _ ! _ ! _ 


5.  Instructions  to  Librarian: 


Priority  (H,  M,  L) 


See  Listing  _ 

Other  (Explain) 


Should  this  module  be  compiled? 


Remainder  of  form  to  be  completed  by  Librarian: 

1.  Is  the  Change  Request  indicated  in  question  1  of  this  form  on  file? 
(If  not,  do  not  make  change.  Return  form  incomplete.) 


2.  Has  question  9  of  the  Change  Request  indicated  in  question  1  here  been  updated  by 

author  of  this  form?  _ 

3.  If  this  is  a  change  to  a  compiled  module  of  design  or  code,  list  any  other  modules 

subsequently  requiring  recompilation:  _ 


Date  Change  completed 


(or  returned  incomplete  _ / _ / _ ) 
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